Abstract:

In order to extract the Chinese frequent strings (CFS) accurately and make better use in language models, a new method for CFS extraction using string segmentation degree is proposed. Unigram and bigram language models based on this CFS extraction method are built. Experiment shows that the CFS based language model can deal with the lack of long distance dependency problem in character and word based language model. It also shows that the CFS based language model has lower model perplexity and higher pinyintocharacter conversion correctness compared with the model based on previous CFS extraction method.

Key words: Chinese frequent string, character distinction degree, string segmentation degree, n-gram language model, pinyin-to-character conversion